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Background 



Under pressure to include more students in large-scale assessment programs, especially those 
students with disabilities, states have used a variety of testing approaches. Several states have 
introduced out-of-level testing as one option for measuring academic proficiency. Out-of-level 
testing, which is most commonly defined as the administration of a test at a grade level below 
the grade a student is enrolled in school, has taken on multiple forms. Of the 17 states that used 
out-of-level tests in school year 2003-2004, some states tested students with disabilities at any 
grade level necessary to reach a student’s level of instruction (VanGetson, Minnema, & Thurlow, 
2004). Other states chose to limit out-of-level testing to only those grade levels available in the 
regular large-scale assessment. States also differed in the test used for below grade level testing. 
Both norm-referenced and criterion-referenced tests were used as well as norm-referenced tests 
with additional criterion-referenced components that augmented the original instrument. 

States’ out-of- level testing policies have been similar in the contentiousness that has surrounded 
them. Proponents of out-of-level testing generally contend that there are three benefits for 
students with disabilities: (1) undue test frustration is avoided, (2) test measurement accuracy 
is improved, and (3) test items are better matched to students’ current educational goals and 
instructional level (Thurlow, Elliott, &Ysseldyke, 1999). It is often thought that testing stu- 
dents with disabilities on the grade level of enrollment in school is actually unfair. Taken to its 
extreme, students can be traumatized by being forced to dwell on test items for which they do 
not have the academic skills. 

On the other hand, opponents of out-of-level testing contend the following: (1) testing students 
below their grade of enrollment does not match the system accountability purpose of a statewide 
assessment, (2) lower grade level testing reflects teachers’, family members’, and students’ 
lower expectations for academic learning, and (3) the resulting instruction over time will focus 
on lower-level standards than those that the student could be striving to acquire (Thurlow et ah, 
1999). By continuing to test a student out of level from one school year to the next, a student 
may lose important opportunities to learn so that the student’s true learning potential may never 
be known. 

Only recently have the many issues that surround out-of-level testing begun to be sorted out in 
the literature. Concerns have arisen about states’ reporting practices since out-level test scores 
are not readily available in states’ data reports that are used for local school improvement plan- 
ning (Minnema & Thurlow, 2003). Determining the prevalence of students with disabilities 
who had been tested out of level on a state by state basis has proved to be difficult (Thurlow, 
Minnema, Bielinski, & Guven, 2003). A focused research study on states’ out-of-level test score 
reporting yielded no state with publicly reported out-of- level test results that were clearly labeled 
and readily identifiable unless state educational agencies produced special data runs of those 
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test data (Minnema & Thurlow, 2003). Recent research findings have demonstrated negative 
instructional effects of testing students with disabilities below grade level (Minnema, Thurlow, 
& Warren, 2004a and 2004b). 

Given the emphasis in current standards-based educational reform efforts for all students to 
receive challenging, grade-level standards-based instruction, these research findings are dis- 
concerting. The research suggests that a relatively large subgroup of students — students with 
disabilities — is receiving instruction that is not on grade level. 

Over the past few years, arguments similar to those raised in the late 1990s have continued to 
circulate in practice among educators and policymakers. The purpose of this research was to 
document the pro and con arguments that surround out-of-level testing. We conducted a survey 
to describe teachers’ and administrators’ perceptions and opinions about out-of- level testing 
when testing students with disabilities in states’ large-scale assessment programs that are used 
for accountability purposes. 



Method -■■■■■■■■■■■■■■■■■■■■■■- 

We created a one page, doubled sided written survey that contained closed and open-ended items 
to describe educators’ perceptions and opinions about out-of- level testing (see Appendix A for 
a copy of the survey instrument). The 16 closed survey items were in the form of opinion state- 
ments requiring respondents to indicate “agree,” “disagree,” “don’t know,” or “no opinion” on 
a 4-point scale. We decided to use this type of scale rather than a Likert scale so that we could 
learn whether respondents knew about out-of- level testing rather than having a measure of the 
strength with which they agreed with the statements in the survey items. The first survey item 
requested a “yes” or “no” response as to whether the participant was familiar with out-of-level 
testing. We also gathered demographic data on four closed survey items and additional opinion 
data on three open-ended items. 

The survey was distributed in four states at state-level teacher and administrator meetings and 
training sessions. We mailed packets of surveys and written directions for completing the sur- 
veys as well as written distribution and collection procedures. One contact person in each state 
educational agency was responsible for distributing and collecting the surveys and returning 
the finished surveys in addressed, pre-paid mailings. In State 1, special education teachers (n 
= 53) and administrators (n = 81) completed surveys. In State 2, only administrators (n = 52) 
participated in the survey. In another state. State 3, both special education teachers (n = 29) and 
district test coordinators (n = 18) returned surveys. The final state. State 4, had special education 
teachers (n = 48), regular education teachers (n = 19), administrators (n = 89), and assessment 
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coordinators (n = 54) participating in the survey. We did not calculate a return rate because our 
sample was a purposive sample where all training attendees were invited to participate, but not 
required to complete a survey. Since attendance was not necessarily recorded at all sessions, we 
had no estimate of the exact number of surveys distributed but not returned. 

All numeric data were entered into spreadsheets and checked for data entry accuracy. We used 
descriptive statistics to analyze these data. The narrative data were transcribed and types of 
responses were tallied for categories of results. A second researcher analyzed one-fifth of the 
narrative data set to ensure accuracy of findings. 

Only descriptive statistics could be used for data analysis of numeric data. Because of this, we set 
decision rules for interpreting our numeric data to draw final conclusions. For those items where 
one category received 40% or more with the other three categories receiving 25% or less of the 
responses, we called this occurrence “a clear majority.” We considered a response pattern to be 
“without a clear majority” when all four categories received 25% or less of the responses. 



Results = 

We present our survey results on a state by state basis by treating each subgroup of our sample 
as a separate data set. For each state, we first describe the state’s sample with various descrip- 
tive statistics. Second, the numeric survey results for each sample subgroup are presented in 
the next tables. Depending on the number of sample subgroups that participated, the next tables 
contain the actual survey items as created for the survey instruments with the states’ test names 
replaced with “state test” to protect the confidentiality of those participating in the research study. 
We also provide a column in our tables to report on the number of “no responses” per survey 
item, which was not part of our original survey instrument (see Appendix A for copies of the 
instruments). Other demographic results are also included in the next few tables to conclude 
the state’s report on numeric findings. Third, as a final section for each state’s survey results, 
we present the results of open-ended survey items. 

Following our presentation of each state’s survey results, we take a step back from our data to 
glean global thematic conclusions that consider all states’ data as a composite whole. While 
these conclusions draw from states with approaches to testing students with disabilities that vary 
widely, our summative ideas are relevant in a general sense for educators and policymakers who 
are pondering the value of this approach to standards-based assessment. 
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state 1 



State 1 is a large western state where only students with disabilities are tested below the grade 
in which they are enrolled in school. The state test is an augmented norm-referenced test that 
contains criterion-referenced items that are aligned to state content standards. State policy allows 
that the State 1 assessment be administered at any level below grade level that is necessary to 
match a student’s level of instruction, but only one or two levels below grade level is considered 
to be a standard test administration. These results are entered into accountability indices at the 
lowest proficiency level at the student’s grade of enrollment. Further below grade level is treated 
as a nonstandard test presentation, which cannot be entered in accountability indices. 

Sample - State 1 

A total of 134 surveys were distributed and returned in State 1. The subgroups in the purposive 
sample from State 1 contained special education teachers (n = 53) and school administrators 
(n = 81) who returned the sample either completed or with an indication that they were not 
familiar with out-of-level testing. Of the participants who returned the surveys, 91% (n = 48) of 
special education teachers and 93% (n = 75) of school administrators responded affirmatively 
to the question about familiarity with out-of-level testing. When asked whether teachers in the 
respondents’ school districts tested students with disabilities out of level, 58% (n = 31) of the 
teachers said “yes,” with 9 teachers not responding. For school administrators, 68% (n = 55) 
indicated that students with disabilities were tested out of level in their school district while 15 
did not respond to this survey item. There were five special education teachers and six school 
administrators who did not respond to the item pertaining to familiarity with out-of-level testing, 
but completed all of the remaining survey items. Overall, we were satisfied that the educators in 
State 1 who responded to our survey were familiar enough with testing students with disabilities 
out of level in large-scale assessments to knowledgeably answer the survey items. 

Other State 1 sample demographic features are presented in Table 1. These data indicated that 
sample participants were well experienced educators. Nearly 75% of teachers and 70% of 
administrators had at least 11 years or more of experience in schools. In terms of educational 
background, the sample was also well educated. Eleven percent of teachers and 7% of adminis- 
trators had Bachelor’s degrees. Almost 90% of the special education teachers and over 90% of 
school administrators held advanced degrees with the majority of both groups holding Master’s 
degrees. Please note that missing responses were not included in this table due to space con- 
straints. In terms of years of experience, seven teachers did not complete this survey with 11 
school administrators not responding. For educational degree information, eight teachers and 
1 1 administrators did not complete the item. 

Numeric Results - State 1 

The numeric results for the two subgroups in State 1 are presented in tables that show all survey 
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Table 1. State 1 Sample - Participant Levels of Experience and Education 



Participants 


Years of Experience^ 


Educational Degree'’ 


1-10 

Years 


11-20 

Years 


21-30 

Years 


31-40 

Years 


Bachelor’s 


Master’s 


Specialist 


Ph.D. 


Special Education 

Teachers 

(n = 53) 


8 

(15%) 


10 

(19%) 


22 

(42%) 


6 

(11%) 


6 

(11%) 


24 

(45%) 


8 

(15%) 


7 

(13%) 




















School 

Administrators 
(n = 81) 


15 

(19%) 


13 

(16%) 


34 

(42%) 


8 

(10%) 


6 

(7%) 


45 

(56%) 


10 

(12%) 


9 

(11%) 



® Non-responses to this item for 7 teachers and 1 1 school administrators. 
Non-responses to this item for 8 teachers and 1 1 school administrators. 



items, on the order presented in the survey. Generally speaking, special education teachers either 
agreed or disagreed with the survey items (sentences). By looking for those items with which 
at least 51% agreed and those items with which at least 51% disagreed, we identified 1 1 out of 
16 closed items (items a-d, f-I, k, n-o). All of these items had relatively low rates of respondents 
indicating no opinion, don’t know, or giving no response. The content of the items suggested a 
pattern in the data of overall favorable opinions about testing students with disabilities out of 
level in states’ large-scale assessments. For instance, nearly two-thirds of the participants agreed 
that out-of-level testing was a good way to include students with disabilities in statewide test- 
ing (item a; 60%). Nearly two-thirds of the respondents disagreed that out-of-level testing had 
negative consequences for all students (item b; 60%) and negative consequences for students 
with disabilities (item c; 58%). In terms of the accuracy of out-of-level test results, 67% of the 
respondents agreed that test data were more accurate when students with disabilities were tested 
below their grade of enrollment in school (item d). 

On those survey items without clear agreement or disagreement (items e, j, 1, m, p) the indeci- 
sion was more often about content that was neutral or positive toward out-of-level testing. For 
example, item e, which suggested that students who participated in out of level tests were more 
motivated test takers, showed 43% agreeing, 12% disagreeing, 15% not knowing, and a total 
of 21% either not responding or having no opinion. Special education teachers were also not 
clearly decisive about whether Individualized Education Program (lEP) teams actually do select 
students accurately for out-of-level tests (item j), whether lEP teams considered future effects 
of out-of-level testing when deciding whether to test out of level or not (item 1), whether parents 
understood the future effects of testing out of level (item m), and whether out-of-level testing 
was easily administered (item p). 
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Table 2. State 1 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level in 
Large-Scale Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing Is a good way to Include students with 
disabilities in the state test. 


32 

(60%) 


12 

(23%) 


0 


2 

(4%) 


7 

(13%) 


B 


Allowing students with disabilities to take the state test 


9 


32 


1 


3 


8 


B 


out of level has negative consequences for all students. 


(17%) 


(60%) 


(2%) 


(6%) 


(15%) 


c. 


Allowing students with disabilities to take the state test out of level 
has negative consequences only for students with disabilities. 


5 

(9%) 


36 

(68%) 


0 


4 

(8%) 


8 

(15%) 


d. 


Students who need to take the state test out of level have more 


30 


11 


1 


4 


7 




accurate test scores than when taking the state test on grade level. 


(57%) 


(21%) 


(2%) 


(8%) 


(13%) 


e. 


Students who need to take the state test out of level are more 


23 


12 


3 


8 


7 




motivated to perform well than when taking the state test on 
grade level. 


(43%) 


(23%) 


(6%) 


(15%) 


(13%) 


f. 


Students who need to take the state test out of level are less 


31 


7 


2 


6 


7 




frustrated than when taking the state test on grade level. 


(58%) 


(13%) 


(4%) 


(11%) 


(13%) 


g- 


Students who need to take the state test out of level guess at 


27 


9 


1 


9 


7 




answers less frequently than when taking the state test on 
grade level. 


(51%) 


(17%) 


(2%) 


(17%) 


(13%) 


h. 


Students who need to take the state test out of level answer more 


28 


7 


2 


9 


7 




test questions than when taking the state test on grade level. 


(53%) 


(13%) 


(4%) 


(17%) 


(13%) 


B 


Individualized Education Program (lEP) teams can select students 


37 


4 


3 


2 


7 


B 


who need out-of-level tests accurately. 


(70%) 


(8%) 


(6%) 


(4%) 


(13%) 


B 


Individualized Education Program flEPt teams actually do select 


21 


13 


1 


10 


8 


B 


students who need out-of-level tests accurately. 


(40%) 


(25%) 


(2%) 


(19%) 


(15%) 


D 


Parents of students with disabilities help make the decision to test 


32 


3 


3 


7 


8 


B 


students out of level in the state test. 


(60%) 


(6%) 


(6%) 


(13%) 


(15%) 


B 


When deciding to test students out of level, Individualized 


22 


9 


3 


11 


8 


1 


Education Program (lEP) teams consider the future consequences 
of taking the state test out of level. 


(42%) 


(17%) 


(6%) 


(21%) 


(15%) 


m. 


Parents of students with disabilities understand the future 


7 


21 


1 


15 


9 




consequences of taking the state test out of level. 


(13%) 


(40%) 


(2%) 


(28%) 


(17%) 


n. 


Students who need to take the state test out of level can meet the 


8 


29 


2 


6 


8 




standards of the grade in which they are enrolled in school. 


(15%) 


(55%) 


(4%) 


(11%) 


(15%) 


0. 


Students who need to take the state test out of level do meet the 


3 


31 


1 


10 


8 




standards of the grade in which they are enrolled in school. 


(6%) 


(58%) 


(2%) 


(19%) 


(15%) 


p- 


Administering the state test out of level is easy to do. 


17 

(32%) 


19 

(36%) 


4 

(8%) 


6 

(11%) 


7 

(13%) 























































































































School administrators’ responses also indicated either agreement or disagreement with the 
survey items (see Table 3). Most agreement or disagreement percentages ranged from 56% to 
78%, although there were more of them with a clear majority (13 of 16 items). Administra- 
tors’ responses in State 1 mirrored those of the special education teacher responses in State 1 
in that the pattern in the responses again revealed favorable opinions about out-of-level testing. 



Table 3. State 1 - School Administrator Opinions on Testing Students with Disabilities Out of Level in Large-Scale 
Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabiiities 
in the state test. 


59 

(73%) 


13 

(16%) 


0 


2 

(2%) 


7 

(9%) 


b. 


Aliowing students with disabilities to take the state test out of levei has 


9 


58 


3 


4 


7 




negative consequences for all students. 


(11%) 


(72%) 


(4%) 


(5%) 


(9%) 


c. 


Aliowing students with disabilities to take the state test out of levei has 


3 


61 


6 


4 


7 




negative consequences only for students with disabilities. 


(4%) 


(75%) 


(7%) 


(5%) 


(9%) 


d. 


Students who need to take the state test out of level have more 


48 


13 


5 


9 


6 




accurate test scores than when taking the state test on grade level. 


(59%) 


(16%) 


(6%) 


(11%) 


(7%) 


e. 


Students who need to take the state test out of level are more 


50 


14 


4 


7 


6 




motivated to perform weil than when taking the state test on grade 
levei. 


(62%) 


(17%) 


(5%) 


(9%) 


(7%) 


f. 


Students who need to take the state test out of level are less 


59 


12 


1 


3 


6 




frustrated than when taking the state test on grade level. 


(73%) 


(15%) 


(1%) 


(4%) 


(7%) 


g- 


Students who need to take the state test out of level guess at answers 


45 


10 


6 


13 


7 




less frequently than when taking the state test on grade level. 


(56%) 


(12%) 


(7%) 


(16%) 


(9%) 


h. 


Students who need to take the state test out of level answer more test 


50 


6 


4 


14 


7 




questions than when taking the state test on grade level. 


(62%) 


(7%) 


(5%) 


(17%) 


(9%) 


■ 


Individualized Education Program (lEP) teams can select students 


63 


4 


3 


5 


6 


who need out-of-level tests accurately. 


(78%) 


(5%) 


(4%) 


(6%) 


(7%) 


■ 


Individualized Education Program (lEP) teams actually do select 


39 


10 


5 


20 


7 


students who need out-of-level tests accurately. 


(48%) 


(12%) 


(6%) 


(25%) 


(9%) 


■ 


Parents of students with disabilities help make the decision to test 


57 


7 


1 


7 


9 


students out of level in the state test. 


(70%) 


(9%) 


(1%) 


(9%) 


(11%) 


n 


When deciding to test students out of level, Individualized Education 


46 


10 


3 


15 


7 


1 


Program (lEP) teams consider the future consequences of taking the 
state test out of level. 


(57%) 


(12%) 


(4%) 


(19%) 


(9%) 


m. 


Parents of students with disabilities understand the future 


19 


25 


1 


28 


8 




consequences of taking the state test out of level. 


(23%) 


(31%) 


(1%) 


(35%) 


(10%) 


n. 


Students who need to take the state test out of level can meet the 


16 


45 


3 


10 


7 




standards of the grade in which they are enrolled in school. 


(20%) 


(56%) 


(4%) 


(12%) 


(9%) 


0. 


Students who need to take the state test out of level do meet the 


3 


49 


4 


18 


7 




standards of the grade in which they are enrolled in school. 


(4%) 


(60%) 


(5%) 


(22%) 


(9%) 


p- 


Administering the state test out of level is easy to do. 


27 

(33%) 


31 

(38%) 


6 

(7%) 


11 

(14%) 


6 

(7%) 
























































































































This was true for all but three of the survey items where there was no clear pattern in the data. 
School administrators’ responses were more evenly spread across response categories as to 
whether lEP teams did select students for out-of-level testing appropriately (item j), whether 
parents understood the future consequences of out-of-level testing for their children (item m), 
and whether out-of-level tests were easy to administer (item p). 

In addition to the educator opinion and demographic data that we gathered with our written 
survey, we also asked what information lEP team members used to select an appropriate grade 
level at which to administer an out-of-level test. Participants were given a list of options with 
the directions to select all types of information that applied to their schools. Participants’ re- 
sponses are displayed in Table 4. The most frequently identified type of student information 
used to select an out-of-level test level was the grade level on which core content instruction 
was delivered. About one third of the participants, both teachers and administrators, indicated 
that students’ previous test scores were used as indicators of appropriate out-of-level test levels 
while approximately 15% of teachers’ and administrators’ responses suggested that standardized 
tests are used. Approximately one-fifth of the responses showed that lEP team members com- 
pare test item content to students’ curricular content to determine a test grade level. Eess than 
5% of all participants responded that a locator test was administered to determine the student’s 
academic grade level at which an out-of-level test should be administered. Approximately 10% 
of respondents indicated another method used to select the out-of-level test grade level. These 
methods included considering student abilities, following eligibility criteria, measuring against 
lEP goals, portfolio assessments, relying on teacher expertise, or following parent requests. 

Table 4. State 1 - Information Used to Choose Out-of-Level Test Grade Levels 





Special Ed 


School 


Student Information Used 


Teachers 


Administrators 


Student’s instructional level 


32 


57 




(60%) 


(70%) 


Student’s previous test scores 


17 


29 




(32%) 


(36%) 


Test company’s locator test scores 


2 


3 




(4%) 


(4%) 


Standardized achievement test score 


7 


14 




(13%) 


(17%) 


Test item and curricular comparison 


8 


18 




(15%) 


(22%) 


Other 


7 


8 




(13%) 


(10%) 
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Narrative Results - State 1 

Participants in State 1 also responded to three open-ended survey items. The subgroups were 
combined across the state for this qualitative analysis. Each of these three items are presented 
in Table 5 with a result statement derived from the most frequent narrative responses and sup- 
portive participant quotes. 

Table 5. State 1 - Narrative Response to Open-ended Survey Items 



Survey Item 


Most Frequent Result 
Statements 


Supportive Quotes 


(1) How does your district use 
out-of-level test scores? 


1a. Out-of-level test scores 
were NOT used in State 1. 


“They don’t use them,’’ “We 
don’t use OOL (out-of-level) 
test scores,” and “The scores 
are not used for anything.” 




1b. Educators did NOT know 
how to use out-of-level test 
scores. 


“1 don’t know” or “Not sure.” 




1c. Out-of-level test scores 
WERE used for monitoring 
student progress. 


“To measure year to year 
growth,” “To measure progress 
of Individual students,” and 
“Comparison of student’s 
results against him/herself.” 




Id. Out-of-level test scores 
were used as raw data ONLY. 


“Only raw scores are 
calculated,” “The state does 
not use out-of-level test 
scores,” and “Teachers can 
only use scores to track a 
student’s progress over time 
without making any other 
comparisons.” 


(2) What are the positive 
aspects or impacts of testing 
students out of level? 


2a. Students are less 
frustrated because they 
experience greater success 
on the out-of-level test than on 
the on-level test. 


“Students feel less frustrated, 
more able,” “Testing students 
out of level can help motivate 
students to succeed,” and 
“Students do not get as 
frustrated when taking the test 
on their instructional level as 
compared to taking the test on 
actual grade enrolled. Student 
tried to answer questions, 
feels good about themselves.” 




2b. Other positive aspects 
were that students felt 
included through their test 
participation, that out-of-level 
tests were more appropriate 
and accurate measurements 
of students’ ability, and that 
students gained valuable test 
taking skills. 
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Table 5. State 1 - Narrative Response to Open-ended Survey Items (continued) 



Survey Item 


Most Frequent Result 
Statements 


Supportive Quotes 


(3) What are the negative 
aspects or impacts of testing 
students out of level? 


3a. Students experienced 
lowered self-esteem when 
testing out of level. 


“Other students see the level 
they are on” and that “Makes 
the student feel inferior.” 




3b. Out-of-level tests did 
NOT produce meaningful 
information. 


“Only raw scores are 
available.” 




3c. Out-of-level testing affected 
students’ academic progress 
negatively. 


“Students cannot achieve 
proficiency on grade of 
enrollment standards.” 




3d. There were NO negative 
aspects or impacts of testing 
students out of level. 


“Even raw scores can be used 
to inform instruction,” and 
“I’m not sure there are any 
negative aspects.” 



State 2 

State 2 is a mid-size southern state that eliminated out-of-level testing in 2003-2004. Prior to 
eliminating out-of-level testing, State 2 allowed out-of-level testing for students with disabili- 
ties who met participation criteria. The state test in State 2 has both criterion-referenced and 
norm-referenced components, but the out-of-level test used only an extended version of the 
norm-referenced component in lieu of the criterion-referenced test. In State 2, students with 
disabilities testing out of level were required to take the reading or math test at least three grade 
levels below the student’s grade of enrollment, and were allowed to take different test levels in 
different content areas. Out-of-level test results in State 2 were aggregated at the student’s grade 
of enrollment at the lowest proficiency level in accountability indices. 

Sample - State 2 

Only school administrators participated in State 2 (n = 52), and 88% of those participants (n = 
46) indicated that they were familiar with out-of-level testing. When asked if teachers in their 
district tested students with disabilities out of level on the state tests, 73% (n = 38) answered “yes” 
while eight school administrators did not respond to this item. Further demographic information 
is presented in Table 6. The majority of the participating school administrators had extensive 
experience, with 65% having 21 or more years of experience. Also, all of the participants hold 
advanced degrees, with the majority (54%) holding Master’s degrees. Missing responses were 
not included in this table; five administrators did not respond to the years of experience ques- 
tion, and four did not respond to the educational degree question. 
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Table 6. State 2 - Participant Levels of Experience and Education 



Participants 


Years of Experience 


Educational Degree 


1-10 

Years 


11-20 

Years 


21-30 

Years 


31-40 

Years 


Bachelor’s 


Master’s 


Specialist 


Ph.D. 


School 

Administrators 
(n = 52) 


3 

(6%) 


7 

(13%) 


27 

(52%) 


7 

(13%) 


0 


28 

(54%) 


13 

(25%) 


7 

(13%) 



Numeric Results - State 2 

The numeric results for the only subgroup in State 2, school administrators, are presented in 
Table 7. Participants showed relative agreement on eight of the survey items in which at least 
50% either agreed or disagreed with the item (items b-c, I, k, n-p). They generally disagreed 
that out-of-level testing has negative consequence for all students (50%) or for students with 
disabilities (54%). In terms of accuracy, 65% said that the out-of- level test results are more ac- 
curate than on- level test results for students with disabilities. Additionally, participants responded 
that students with disabilities who are tested out of level cannot (65%) and do not (73%) meet 
grade of enrollment standards. Finally, 71% answered that administering out-of-level tests is 
not easy to do. The pattern that emerges in these data depicts questionable opinions of out-of- 
level testing. 

On eight of the items, participants showed no clear pattern of response. This included school 
administrators’ opinion on whether out-of-level testing is a good way to include students with 
disabilities in the state tests (item a). There was striking disagreement between respondents on 
whether lEP teams accurately select students who need to test out of level, with 38% agree- 
ing, 42% disagreeing, 6% not knowing, and 14% with no opinion or not responding (item j), 
or whether lEP teams consider the future consequence of testing out of level when making this 
decision, with 44% agreeing, 37% disagreeing, 4% not knowing, and 16% with no opinion or 
not responding (item 1). Also, school administrators were not visibly decisive about students’ 
motivation (item e), frustration (item f), amount of guessing (item g), and amount of questions 
answered when they are tested out of level (item h), or about whether parents understand the 
future consequences for their child who is tested out of level (item m). 

We also asked school administrators in State 2 what information lEP teams use to select the ap- 
propriate test grade level at which to administer an out-of-level test to a student with a disability. 
Participants were given a list of options with the directions to select all types of information 
that applied to their school. Responses are displayed in Table 8. School administrators indicated 
that the student’s instructional level and the student’s previous test scores are the most prevalent 
methods for selecting the out-of- level test grade level. Some participants (21%) provided another 
method of selection not included in the list. These methods included considering the student’s 
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age, grade level, prior lEP, present lEP, academic performance and ability level, as well as the 
state eligibility criteria or a combination of all of the listed methods. 



Table 7. State 2 - School Administrators Opinions on Testing Students with Disabilities Out of Level 
in Large-Scale Assessments 



■ 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


Don’t 

Know 


a. 


Out-of-level testing is a good way to include students with 
disabilities in our state tests. 


18 

(35%) 


25 

(48%) 


1 

(2%) 


2 

(4%) 


6 

(12%) 


b. 


Aliowing students with disabilities to take the state tests out of 
levei has negative consequences for aN students. 


15 

(29%) 


26 

(50%) 


2 

(4%) 


3 

(6%) 


6 

(12%) 


c. 


Aliowing students with disabilities to take the state tests out 
of ievel has negative consequences only for students with 
disabilities. 


13 

(25%) 


28 

(54%) 


0 


3 

(6%) 


8 

(15%) 


d. 


Students who need to take the state tests out of levei have more 
accurate test scores than when taking the state tests on grade 
levei. 


7 

(13%) 


34 

(65%) 


2 

(4%) 


3 

(6%) 


6 

(12%) 


e. 


Students who need to take the state tests out of levei are more 
motivated to perform weil than when taking the state tests on 
grade ievel. 


12 

(23%) 


24 

(46%) 


4 

(8%) 


5 

(10%) 


7 

(13%) 


f. 


Students who need to take the state tests out of level are less 
frustrated than when taking the state tests on grade level. 


25 

(48%) 


16 

(31%) 


2 

(4%) 


1 

(2%) 


8 

(15%) 


g- 


Students who need to take the state tests out of level guess at 
answers less frequently than when taking the state tests on 
grade level. 


17 

(33%) 


16 

(31%) 


4 

(8%) 


9 

(17%) 


6 

(12%) 


h. 


Students who need to take the state tests out of level answer 
more test questions than when taking the state tests on grade 
level. 


18 

(35%) 


12 

(23%) 


4 

(8%) 


11 

(%) 


7 

(13%) 


■ 


Individualized Education Program (lEP) teams can select 
students who need out-of-level tests accurately. 


27 

(52%) 


16 

(31%) 


0 


3 

(6%) 


6 

(12%) 


■ 


Individualized Education Prooram flEPf teams actually do select 
students who need out-of-level tests accurately. 


20 

(38%) 


22 

(42%) 


1 

(2%) 


3 

(6%) 


6 

(12%) 


■ 


Parents of students with disabilities help make the decision to 
test students out of level in the State tests. 


34 

(65%) 


11 

(21%) 


1 

(2%) 


0 


6 

(12%) 


1 


When deciding to test students out of level, Individualized 
Education Program (lEP) teams consider the future 
consequences of taking the state tests out of level. 


23 

(44%) 


19 

(37%) 


2 

(4%) 


2 

(4%) 


6 

(12%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


15 

(29%) 


22 

(42%) 


3 

(6%) 




7 

(13%) 


n. 


Students who need to take the state tests out of level can meet 
the standards of the grade in which they are enrolled in school. 


n 


34 

(65%) 


1 

(2%) 


4 

(8%) 


7 

(13%) 


0. 


Students who need to take the state tests out of level do meet 
the standards of the grade in which they are enrolled in school. 


4 

(8%) 


38 

(73%) 


1 

(2%) 


3 

(6%) 


6 

(12%) 


p- 


Administering the state tests out of level is easy to do. 


6 

(12%) 


37 

(71%) 


2 

(4%) 


1 

(2%) 


6 

(12%) 






















































































































Table 8. State 2 - Student Information Used to Choose Out-of-Level Test Grade Levels 



Student Information Used 


School 

Administrators 


Students’ instructional level 


35 

(67%) 


Students’ previous test scores 


29 

(56%) 


Test company’s locator test scores 


1 

(2%) 


Test item and curricular comparison 


0 


Other 


11 

(21%) 



Narrative Results - State 2 

The school administrators in State 2 also responded to three open-ended survey items. Those 
results are in Table 9. 

Table 9. State 2 - Narrative Response to Open-ended Survey Items 



Survey Items 


Most Frequent Result 
Statements 


Supportive Quotes 


(1) How does your district use 
out-of-level test scores? 


la. Out-of-level testing allowed 
more students with disabilities 
to participate in statewide 
testing. 


“To fulfill state requirements 
for everybody participating in 
assessment.” 




1b. Out-of-level test results 
showed students’ academic 
progress. 


“Measure growth of individual 
students from one year to the 
next,” and “Look at student’s 
area of weakness and provide 
instruction for weakness.” 




1c. Out-of-level test results 
were NOT used or NOT known 
how to use. 


“We don’t,” and “The results 
are meaningless, not valid.” 


(2) What are the positive 
aspects or impacts of testing 
students out of level? 


2a. Students were less 
frustrated when taking out-of- 
level tests. 


“Students are less frustrated.” 




2b. Students were more 
motivated to finish an out-of- 
level test. 


“Students are more willing to 
seriously complete the test.” 




2c. Out-of-level tests were a 
more appropriate, accurate 
assessment of student ability. 


“Students are challenged more 
appropriately.” 




2d. Students’ scores 
COULD be included in the 
accountability system. 


“Less of a negative impact on 
district accountability score.” 
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Table 9. State 2 - Narrative Response to Open-ended Survey Items (continued) 



Survey Items 


Most Frequent Result 
Statements 


Supportive Quotes 




2e. There were NO positive 
aspects or impacts of testing 
students out of level. 




(3) What are the negative 
aspects or impacts of testing 
students out of level? 


3a. Students would NOT be 
able to obtain a high school 
diploma. 


“Requirement that takes out 
of level participants off the 
regular diploma track.” 




3b. The results were NOT 
useful. 


“Scores are of little to zero 
usefulness.” 




3c. Students were held to 
LOWER learning expectations. 


“Possibility of lowered 
expectations once again by 
teachers.” 




3d. There were problems in 
administering the out-of-level 
tests. 


“It is time consuming to plan 
for and to administer.” 



State 3 

State 3 is a small eastern state that also eliminated out-of-level testing in 2003-2004. The state’s 
testing program included a combination of both criterion-referenced and norm-referenced exams. 
Only students with disabilities in grades 5, 8, and 10 were eligible to test out of level in the 
state’s testing program, and were allowed to take only a grade 3, 5, or 8 test. Like State 2, State 
3 included out-of- level test results at the student’s grade of enrollment at the lowest proficiency 
in accountability indices. 

Sample - State 3 

Two subgroups returned a total of 47 surveys in State 3: special education teachers (n = 29) 
and district test coordinators (n = 18). The number of respondents in this state was lower than 
in other states. It should be noted that only 18 district test coordinators participated in this 
study in State 3. Because of the low number of test coordinators’ in particular, caution must 
be exercised in interpreting these findings. Special education teachers either received their sur- 
veys at the training session (n = 9) or in the mail (n = 20). No differences were found between 
the two data sets when analyzed separately; therefore, the results were combined to form one 
subgroup of special education teachers. Of the special education teachers, 93% (n = 27) were 
familiar with out-of-level testing; 100% (n = 18) of the district test coordinators were familiar 
with out-of-level testing. The two special education teacher participants who were not familiar 
with out-of-level testing did not complete the rest of the survey. Only 26% (n = 7) of the special 
education teachers and 11% (n = 2) of the district test coordinators indicated that teachers in 
their districts use out-of-level tests to assess students with disabilities in the state tests. (Three 
special education teachers and seven district test coordinators did not respond.) 
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Table 10 shows the years of experience and educational degree of the participants. The special 
education teachers and district test coordinators were both very experienced in the field, with 
76% of special education teachers and 72% of district test coordinators having 1 1 or more years 
of experience. Also, 100% percent of both subgroups have advanced degrees, with the major- 
ity (69% of special education teachers and 56% of district test coordinators) holding Master’s 
degrees. There were four nonresponses to the years of experience item and three nonresponses 
to the educational degree item for the special education teachers. There were four nonresponses 
for both items for the district test coordinators. 



Table 10. State 3 - Participant Levels of Experience and Education 



Participants 


Years of Experience 


Educational Degree 


1-10 

Years 


11-20 

Years 


21-30 

Years 


31-40 

Years 


Bachelor’s 


Master’s 


Specialist 


Ph.D. 


Special Education 

Teachers 

(n = 29) 


2 

(7%) 


4 

(14%) 


17 

(59%) 


1 

(3%) 


0 


20 

(69%) 


3 

(10%) 


3 

(10%) 


District Test 
Coordinator 
(n = 18) 


1 

(6%) 


4 

(22%) 


7 

(39%) 


2 

(11%) 


0 


10 

(56%) 


1 

(6%) 


3 

(17%) 



Numeric Results - State 3 

The numeric results for the special education teachers and district test coordinators in State 3 are 
presented in Tables 11 and 12, respectively. Special education teachers were in relative agree- 
ment with each other on half (eight) of the survey items. For instance, more than two-thirds of 
these participants agreed that students are less frustrated when allowed to take an out-of-level 
test (item f; 66%), lEP teams can accurately select students in need of testing out of level (item 
i; 66%), students who take out-of-level tests do not meet their grade of enrollment standards 
(item o; 69%), and administering out-of-level tests is not easy to do (item p; 72%). Special 
education teachers indicated less agreement on the other half of the items. Specifically, there 
was disagreement on whether out-of-level testing is a good way to include students with dis- 
abilities in states’ tests (item a), provides more accurate test scores than an on-level test (item 
d), or increases student test performance motivation (item e). The lack of agreement on some of 
the items may have been a result of many participants not knowing the answer to the question. 
A relatively high number of participants answered “Don’t know” on items g (31%), h (34%), 
and m (31%). 
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Table 11. State 3 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level in 
Large-Scale Assessments 



■ 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabilities 
in our state tests. 


11 

(38%) 


12 

(41%) 


0 


1 

(3%) 


5 

(17%) 


b. 


Allowing students with disabilities to take the state tests out of level 
has negative consequences for aH students. 


10 

(34%) 


13 

(45%) 


H 


1 

(3%) 


2 

(7%) 


c. 


Allowing students with disabilities to take the state tests out of level 
has negative consequences only for students with disabilities. 


9 

(31%) 


16 

(55%) 


2 

(7%) 


0 


2 

(7%) 


d. 


Students who need to take the state tests out of level have more 
accurate test scores than when taking the state tests on grade level. 


12 

(41%) 


9 

(31%) 


0 




3 

(10%) 


e. 


Students who need to take the state tests out of level are more 
motivated to perform well than when taking the state tests on grade 
level. 


10 

(34%) 


9 

(31%) 


3 

(10%) 


4 

(14%) 


3 

(10%) 


f. 


Students who need to take the state tests out of level are less 
frustrated than when taking the state tests on grade level. 


19 

(66%) 


4 

(14%) 


0 


4 

(14%) 


2 

(7%) 


g- 


Students who need to take the state tests out of level guess at 
answers less frequently than when taking the state tests on grade 
level. 


14 

(48%) 


1 

(3%) 


3 

(10%) 


9 

(31%) 


2 

(7%) 


h. 


Students who need to take the state tests out of level answer more 
test questions than when taking the state tests on grade level. 


12 

(41%) 


2 

(7%) 


3 

(10%) 


10 

(34%) 


2 

(7%) 


■ 


Individualized Education Program (lEP) teams can select students 
who need out-of-level tests accurately. 


19 

(66%) 


4 

(14%) 


0 


3 

(10%) 


2 

(7%) 


■ 


Individualized Education Proaram (lEP) teams actually do select 
students who need out-of-level tests accurately. 


14 

(48%) 


5 

(17%) 


2 

(7%) 


6 

(21%) 


2 

(7%) 


■ 


Parents of students with disabilities help make the decision to test 
students out of level in the state tests. 


17 

(59%) 


5 

(17%) 


0 


5 

(17%) 


2 

(7%) 


1 


When deciding to test students out of level, Individualized Education 
Program (lEP) teams consider the future consequences of taking 
the state tests out of level. 


16 

(55%) 


2 

(7%) 


1 

(3%) 


7 

(24%) 


3 

(10%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


3 

(10%) 


12 

(41%) 


1 

(3%) 


9 

(31%) 


4 

(14%) 


n. 


Students who need to take the state tests out of level can meet the 
standards of the grade in which they are enrolled in school. 


5 

(17%) 


15 

(52%) 


0 


3 

(10%) 


6 

(21%) 


0. 


Students who need to take the state tests out of level do meet the 
standards of the grade in which they are enrolled in school. 


0 


20 

(69%) 


0 


6 

(21%) 


3 

(10%) 


p- 


Administering the state tests out of level is easy to do. 


2 

(7%) 


21 

(72%) 


0 


4 

(14%) 


2 

(7%) 



The district test coordinators appeared to show more of a pattern of agreement than did the 
special education teachers (see Table 12). On eight of the items, the majority of the district test 
coordinators either agreed or disagreed with the item; for five of those items there was over 
60% agreement or disagreement, depending on the item. Participants did not agree that testing 
out of level has negative consequences for all students (item b; 67%). They agreed that students 
testing out of level are less frustrated (item f ; 6 1 %) and guess less frequently (item g; 6 1 %) than 
if they took an on-level test, and they agreed that lEP teams can appropriately select students 
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in need of testing out of level (item i; 67%) and parents help make this decision (item k; 61%). 
Additionally, on two of the items (h and o), there was unanimous agreement or disagreement 
among those who responded and had an opinion. The district test coordinators agreed that stu- 
dents answer more questions when tested out of level as opposed to on level, and disagreed that 
students who are tested out of level do meet their grade of enrollment standards. It should be 
noted that the percentage of respondents who did not know the answer to these two questions 
was relatively high. 

There was no pattern of agreement or disagreement among the district test coordinators on six of 
the items. Most notably, participants as a group were unsure about the accuracy of out-of-level 
test scores in comparison to on-level test scores (item d), the student’s motivation to perform 
on out-of-level tests as opposed to on-level tests (item e), whether parents understand the con- 
sequences of testing their child out of level (item m), and whether administering out-of-level 
tests is easy to do (item p). It should be noted that only 18 district test coordinators participated 
in this study in State 3; the low number of participants should be kept in mind when interpret- 
ing these data. 

Special education teachers and district test coordinators in State 3 were also asked to provide what 
information lEP teams in their school district use to select the appropriate test grade level at which 
to administer an out-of-level test to a student with a disability. Participants were given a list of 
options with the directions to select all types of information that applied to their school district. 
Responses are displayed in Table 13. Both subgroups indicated that the student’s instructional 
level and the student’s previous test scores are the most prevalent methods for selecting the out- 
of- level test grade level. Test item and curricular comparison was also used in the participants’ 
schools. Some participants provided another method of selection not included in the list. These 
methods included psychological and achievement testing, considering the overall functioning 
of the student and classroom achievement, weighing the consequences of testing out of level for 
that student, relying on state mandated criteria, and considering individual student needs. 

Narrative Results - State 3 

Participants in State 3 also responded to open-ended survey items. The subgroups were combined 
across the state for this qualitative analysis. Results from our content analysis are in Table 14. 
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Table 12. State 3 - District Test Coordinators Opinions on Testing Students with Disabilities Out of Level in 
Large-Scale Assessments 





Survey Items 


No 

Response 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


a. 


Out-of-level testing is a good way to include students with 
disabilities in our state tests. 


2 

(11%) 


9 

(50%) 


4 

(22%) 


1 

(6%) 


2 

(11%) 


b. 


Allowing students with disabilities to take the state tests out of 
level has negative consequences for aN students. 


2 

(11%) 


3 

(17%) 


12 

(67%) 


0 


1 

(6%) 


c. 


Allowing students with disabilities to take the state tests out 
of level has negative consequences only for students with 
disabilities. 


2 

(11%) 


3 

(17%) 


9 

(50%) 


2 

(11%) 


2 

(11%) 


d. 


Students who need to take the state tests out of level have 
more accurate test scores than when taking the state tests on 
grade level. 


2 

(11%) 


7 

(39%) 


3 

(17%) 


2 

(11%) 


4 

(22%) 


e. 


Students who need to take the state tests out of level are more 
motivated to perform well than when taking the state tests on 
grade level. 


2 

(11%) 


7 

(39%) 


5 

(28%) 


1 

(6%) 


3 

(17%) 


f. 


Students who need to take the state tests out of level are less 
frustrated than when taking the state tests on grade level. 


2 

(11%) 


11 

(61%) 


1 

(6%) 


1 

(6%) 


3 

(17%) 


g- 


Students who need to take the state tests out of level guess at 
answers less frequently than when taking the state tests on 
grade level. 


2 

(11%) 


11 

(61%) 


1 

(6%) 


0 


4 

(22%) 


h. 


Students who need to take the state tests out of level answer 
more test questions than when taking the state tests on grade 
level. 


2 

(11%) 


8 

(44%) 


0 


3 

(17%) 


5 

(28%) 


1 


Individualized Education Program (lEP) teams can select 
students who need out-of-level tests accurately. 


2 

(11%) 


12 

(67%) 


1 

(6%) 


0 


3 

(17%) 


1 


Individualized Education Prooram (lEPf teams actuailv do 
select students who need out-of-level tests accurately. 


2 

(11%) 


8 

(44%) 


2 

(11%) 


0 


6 

(33%) 


1 


Parents of students with disabilities help make the decision to 
test students out of level in the state tests. 


2 

(11%) 


11 

(61%) 


2 

(11%) 


0 


3 

(17%) 


1 


When deciding to test students out of level, Individualized 
Education Program (lEP) teams consider the future 
consequences of taking the state tests out of level. 


2 

(11%) 


10 

(56%) 


1 

(6%) 


0 


5 

(28%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


2 

(11%) 


4 

(22%) 


4 

(22%) 


1 

(6%) 


7 

(39%) 


n. 


Students who need to take the state tests out of level can meet 
the standards of the grade In which they are enrolled In school. 


2 

(11%) 


1 

(6%) 


6 

(33%) 


1 

(6%) 


8 

(%) 


0. 


Students who need to take the state tests out of level do meet 
the standards of the grade In which they are enrolled In school. 


2 

(11%) 


0 


8 

(44%) 


1 

(6%) 


7 

(39%) 


p- 


Administering the state tests out of level is easy to do. 


2 

(11%) 


5 

(28%) 


6 

(33%) 


1 

(6%) 


4 

(22%) 
























































































































Table 13. State 3 - Student Information Used to Choose Out-of-Level Test Grade Levels 





Special Ed 


District Test 


Student Information Used 


Teachers 


Coordinators 


Student’s instructional level 


15 


9 




(52%) 


(50%) 


Student’s previous test scores 


8 


6 




(28%) 


(33%) 


Test company’s locator test scores 


0 


1 






(6%) 


Test item and curricular comparison 


4 


3 




(14%) 


(17%) 


Other 


11 


3 




(38%) 


(17%) 



Table 14. Narrative Response to Open-ended Survey Items for State 3 



Survey Items 


Most Frequent Result 
Statements 


Supportive Quotes 


(1) How does your district use 
out-of-level test scores? 


la. Out-of-level test scores 
were used for student 
progress monitoring. 


“To measure individual 
students’ progression in 
general education curriculum.” 




1 b. The use of out-of-level test 
scores is NOT known. 




(2) What are the positive 
aspects or impacts of testing 
students out of level? 


2a. Out-of-level tests increased 
student self-esteem and 
decrease student frustration. 


“Students are not so frustrated 
with concepts because they 
are not completely unfamiliar,” 
and “Students feel more 
capable with increased 
success.” 




2b. There were no positive 
aspects or impacts of testing 
students with disabilities out of 
level. 




(3) What are the negative 
aspects or impacts of testing 
students out of level? 


3a. Future consequences 
included the possibility of 
NOT receiving a high school 
diploma. 


“Eliminates [the] opportunity 
for [an] academic diploma.” 



State 4 

State 4 is a small New England state where any student may take the state tests out of level as 
long as that student receives approval from the Student Support Team, which is located in the 
state educational agency. The state uses only criterion-referenced exams in its state tests. Students 
testing out of level are only allowed to test at the grade levels offered in the regular assessment 
(grade 4, 8, and 10). Out-of- level test scores are transformed into an on-grade level equivalent 
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using score transformation rules, but out-of-level test scores are automatically entered at the 
lowest proficiency level at the student’s grade of enrollment for accountability indices. 

Sample - State 4 

State 4 had special education teachers (n = 48), regular education teachers (n = 19), administra- 
tors (n = 89), and assessment eoordinators (n = 54) participate in the survey for a total of 210 
responses. A total of 88% (n = 42) of the special education teachers, 68% (n = 13) of the regular 
education teachers, 91% (n = 81) of the administrators, and 85% (n = 46) of the assessment 
coordinators indicated that they were familiar with out-of-level testing. When asked whether 
teachers in their school districts used out-of-level testing, 75% (n = 36, 9 nonresponses) of the 
special education teachers, 37% (n = 7, 9 nonresponses) of the regular education teachers, 82% 
(n = 73, 12 nonresponses) of the administrators, and 76% (n = 41, 9 nonresponses) of the as- 
sessment coordinators answered “yes.” 

We asked participants for demographic information pertaining to their experience as educators 
and the highest educational degree obtained. This information is presented in Table 15. Overall, 
special education and regular education teachers were less experienced than assessment coor- 
dinators and school administrators. The largest respondent group for both the special education 
teachers (35%) and the regular education teachers (32%) had 10 years or less of teaching experi- 
ence. The largest respondent group for the assessment coordinators had 1 1-20 years of experience 
(33%), indieating slightly more experience than the teachers. And the largest respondent group 
for the school administrators had 21-30 years of experience (39%). However, school adminis- 
trators had the most experience of all of the groups, with the majority (57%) having 21 or more 
years of experience in education. The most common degree held for all groups was a Master’s 
degree, with 50% of the special education teachers, 37% of the regular education teachers. 



Table 15. State 4 - Participant Levels of Experience and Education 



Participants 


Years of Experience 


Educational Degree 


1-10 

Years 


11-20 

Years 


21-30 

Years 


31-40 

Years 


Bachelor’s 


Master’s 


Specialist 


Ph.D. 


Special Education 

Teachers 

(n = 48) 


17 

(35%) 


13 

(27%) 


9 

(19%) 


3 

(6%) 


13 

(27%) 


24 

(50%) 


6 

(13%) 


0 


Regular Education 

Teachers 

(n = 19) 


6 

(32%) 


4 

(21%) 


3 

(16%) 


0 


6 

(32%) 


7 

(37%) 


0 


0 


School 

Administrators 
(n = 89) 


3 

(3%) 


24 

(27%) 


35 

(39%) 


16 

(18%) 


1 

(1%) 


60 

(67%) 


15 

(17%) 


2 

(2%) 


Assessment 
Coordinators 
(n = 54) 


13 

(24%) 


18 

(33%) 


11 

(20%) 


1 

(2%) 


9 

(27%) 


27 

(50%) 


4 

(7%) 


1 

(2%) 
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67% of the school administrators, and 50% of the assessment coordinators holding this degree. 
Nonresponses were not included in the table due to space limitations; five special education 
teachers, six regular education teachers, and 1 1 school administrators did not respond to these 
two items. Additionally, 10 assessment coordinators did not respond to the years of experience 
item, and 13 assessment coordinators did not respond to the educational degree item. 

Numeric Results - State 4 

The special education teachers showed exceptional consistency in their responses, which are 
displayed in Table 16. The respondents came to a majority consensus (over 52% of respon- 
dents answered in the same manner) on 14 of the 16 items (all but items 1 and m). For the most 
part, special education teachers in State 4 had a very positive opinion of out-of-level testing, 
as evidenced by 83% agreeing that out-of-level testing is a good way to include students with 
disabilities in the state tests. The only negative sentiment emerged when questioned about 
meeting grade of enrollment standards; participants responded that students cannot and do not 
achieve proficiency on grade of enrollment standards (items n and o). The two items on which 
the special education teachers had no clear response pattern focused on lEP teams considering 
the future consequences of testing out of level when making that decision for a student (item 1), 
and parents understanding the consequences of testing their child out of level (item m). 

The regular education teachers’ responses showed less of a consistent pattern than did the special 
education teachers’ responses. As illustrated in Table 17, only six items elicited a clear response 
pattern from the regular education teachers. They answered that out-of-level testing is a good 
way to include students with disabilities in the state tests (item a; 68%), does not have negative 
consequences for students (item b; 68%) or only students with disabilities (item c; 58%), yields 
more accurate scores than on-level test scores (item d; 53%), and produces less student frustra- 
tion than on-level tests (item f; 53%). They also responded that lEP teams can accurately select 
students in need of out-of-level testing (item i; 68%). There were some items on which regular 
education teachers exhibited a pattern of response (e.g., item j) without a majority response 
either in agreement or disagreement with the survey items. There was a high no response rate 
(32%) in this already small group of participants, and this may have had an effect on the response 
frequencies for these items. The only two items on which the regular education teachers in State 
4 were undecided were item m (if parents understand the consequences of testing their child 
out of level) and item p (if administering an out-of-level test is easy to do). 

Much like the special education teachers, the school administrators (see Table 18) and assessment 
coordinators (see Table 19) showed a strong pattern of consistency in answering the survey items. 
Eor the school administrators, a clear pattern of response was noted in 14 of the 16 items; this 
was true for 13 of the 16 items for the assessment coordinators, almost all with positive opin- 
ions of out-of-level testing. Similar to the special education teachers, the school administrators 
and assessment coordinators responded that students cannot and do not achieve proficiency on 
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Table 16. State 4 - Special Education Teacher Opinions on Testing Students with Disabilities Out of Level 
in Large-Scale Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabiiities in 
the state tests. 


40 

(83%) 


1 

(2%) 


1 

(2%) 


1 

(2%) 


5 

(10%) 


b. 


Aiiowing students with disabilities to take the state tests out of levei has 
negative consequences for aN students. 


1 

(2%) 


37 

(77%) 


3 

(6%) 


2 

(4%) 


5 

(10%) 


c. 


Aiiowing students with disabilities to take the state tests out of levei has 
negative consequences only for students with disabilities. 


1 

(2%) 


36 

(75%) 


4 

(8%) 


2 

(4%) 


5 

(10%) 


d. 


Students who need to take the state tests out of level have more accurate 
test scores than when taking state tests on grade ievei. 


39 

(81%) 


2 

(4%) 


1 

(2%) 


1 

(2%) 


5 

(10%) 


e. 


Students who need to take the state tests out of level are more motivated 
to perform weil than when taking state tests on grade ievei. 


31 

(65%) 


5 

(10%) 


2 

(4%) 


5 

(10%) 


5 

(10%) 


f. 


Students who need to take the state tests out of level are less frustrated 
than when taking state tests on grade level. 


41 

(85%) 


1 

(2%) 


0 


1 

(2%) 


5 

(10%) 


g- 


Students who need to take the state tests out of level guess at answers 
less frequently than when taking state tests on grade level. 


29 

(60%) 


5 

(10%) 


0 


8 

(17%) 


6 

(13%) 


h. 


Students who need to take the state tests out of level answer more test 
questions than when taking state tests on grade level. 


39 

(81%) 


0 


0 


4 

(8%) 


5 

(10%) 


1 


Individualized Education Program (lEP) teams can select students who 
need out-of-level tests accurately. 


41 

(85%) 


0 


0 


2 

(4%) 


5 

(10%) 


1 


Individualized Education Prooram flEPf teams actually do select students 
who need out-of-level tests accurately. 


37 

(77%) 


1 

(2%) 


1 

(2%) 


4 

(8%) 


5 

(10%) 


1 


Parents of students with disabilities help make the decision to test 
students out of level in the state tests. 


33 

(69%) 


3 

(6%) 


1 

(2%) 


5 

(10%) 


6 

(13%) 


1 


When deciding to test students out of level, Individualized Education 
Program (lEP) teams consider the future consequences of taking the 
state tests out of level. 


20 

(42%) 


5 

(10%) 


5 

(10%) 


13 

(27%) 


5 

(10%) 


m. 


Parents of students with disabilities understand the future consequences 
of taking the state tests out of level. 


8 

(17%) 


7 

(15%) 


3 

(6%) 


24 

(50%) 


6 

(13%) 


n. 


Students who need to take the state tests out of level can meet the 
standards of the grade in which they are enrolled in school. 


9 

(19%) 


25 

(52%) 


1 

(2%) 


8 

(17%) 


5 

(10%) 


0. 


Students who need to take the state tests out of level do meet the 
standards of the grade in which they are enrolled in school. 


5 

(10%) 


27 

(56%) 


1 

(2%) 


10 

(21%) 


5 

(10%) 


p- 


Administering the state tests out of level is easy to do. 


26 

(54%) 


3 

(6%) 


4 

(8%) 


9 

(19%) 


6 

(13%) 
























































































































Table 17. State 4 - Regular Education Teacher Opinions on Testing Students with Disabilities Out 
of Level in Large-Scale Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabiiities in 
the state tests. 


13 

(68%) 


0 


0 


0 


6 

(32%) 


b. 


Aiiowing students with disabilities to take the state tests out of levei has 
negative consequences for aN students. 


0 


13 

(68%) 


0 


0 


6 

(32%) 


c. 


Aiiowing students with disabilities to take the state tests out of levei has 
negative consequences only for students with disabilities. 


1 

(5%) 


11 

(58%) 


1 

(5%) 


0 


6 

(32%) 


d. 


Students who need to take the state tests out of ievel have more 
accurate test scores than when taking state tests on grade ievei. 


10 

(53%) 


1 

(5%) 


0 


2 

(11%) 


6 

(32%) 


e. 


Students who need to take the state tests out of ievel are more 
motivated to perform weil than when taking state tests on grade levei. 


8 

(42%) 


2 

(11%) 


1 

(5%) 


2 

(11%) 


6 

(32%) 


f. 


Students who need to take the state tests out of ievel are less 
frustrated than when taking the state tests on grade level. 


10 

(53%) 


2 

(11%) 


0 


1 

(5%) 


6 

(32%) 


g- 


Students who need to take the state tests out of level guess at answers 
less frequently than when taking state tests on grade level. 


8 

(42%) 


1 

(5%) 


1 

(5%) 


3 

(16%) 


6 

(32%) 


h. 


Students who need to take the state tests out of level answer more test 
questions than when taking state tests on grade level. 


9 

(47%) 


1 

(5%) 


2 

(11%) 


1 

(5%) 


6 

(32%) 


1 


Individualized Education Program (lEP) teams can select students who 
need out-of-level tests accurately. 


13 

(68%) 


0 


0 


0 


6 

(32%) 


1 


Individualized Education Prooram flEPf teams actually do select 
students who need out-of-level tests accurately. 


8 

(42%) 


0 


2 

(11%) 


3 

(16%) 


6 

(32%) 


1 


Parents of students with disabilities help make the decision to test 
students out of level in the state tests. 


8 

(42%) 


2 

(11%) 


0 


3 

(16%) 


6 

(32%) 


1 


When deciding to test students out of level, Individualized Education 
Program (lEP) teams consider the future consequences of taking the 
state tests out of level. 


9 

(47%) 


2 

(11%) 


1 

(5%) 


1 

(5%) 


6 

(32%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


3 

(16%) 


4 

(21%) 


1 

(5%) 


5 

(26%) 


6 

(32%) 


n. 


Students who need to take the state tests out of level can meet the 
standards of the grade in which they are enrolled in school. 


1 

(5%) 


7 

(37%) 


3 

(16%) 


2 

(11%) 


6 

(32%) 


0. 


Students who need to take the state tests out of level do meet the 
standards of the grade in which they are enrolled in school. 


1 

(5%) 


9 

(47%) 


0 


3 

(16%) 


6 

(32%) 


p- 


Administering the state tests out of level is easy to do. 


5 

(26%) 


3 

(16%) 


2 

(11%) 


3 

(16%) 


6 

(32%) 
























































































































Table 18. State 4 -School Administrator Opinions on Testing Students with Disabilities Out of Level in Large- 
Scale Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabilities In 
the state tests. 


75 

(84%) 


2 

(2%) 


2 

(2%) 


1 

(1%) 


9 

(10%) 


1 


Allowing students with disabilities to take the state tests out of level has 
negative consequences for aH students. 


4 

(4%) 


68 

(76%) 


4 

(4%) 


4 

(4%) 


9 

(10%) 


c. 


Allowing students with disabilities to take the state tests out of level has 
negative consequences only for students with disabilities. 


4 

(4%) 


66 

(74%) 


4 

(4%) 


5 

(6%) 


10 

(11%) 


d. 


Students who need to take the state tests out of level have more 
accurate test scores than when taking state tests on grade level. 


59 

(66%) 


10 

(11%) 


3 

(3%) 


9 

(10%) 


8 

(9%) 


e. 


Students who need to take the state tests out of level are more 
motivated to perform well than when taking state tests on grade level. 


46 

(52%) 


16 

(18%) 


3 

(3%) 


15 

(17%) 


9 

(10%) 


f. 


Students who need to take the state tests out of level are less frustrated 
than when taking state tests on grade level. 


70 

(79%) 


3 

(3%) 


3 

(3%) 


5 

(6%) 


8 

(9%) 


g- 


Students who need to take the state tests out of level guess at answers 
less frequently than when taking state tests on grade level. 


48 

(54%) 


2 

(2%) 


1 

(1%) 


30 

(34%) 


8 

(9%) 


h. 


Students who need to take the state tests out of level answer more test 
questions than when taking state tests on grade level. 


56 

(63%) 


1 

(1%) 


1 

(1%) 


23 

(26%) 


8 

(9%) 


1 


Individualized Education Program (lEP) teams can select students who 
need out-of-level tests accurately. 


79 

(89%) 


0 


1 

(1%) 


1 

(1%) 


8 

(9%) 


1 


Individualized Education Prooram flEPf teams actually do select 
students who need out-of-level tests accurately. 


68 

(76%) 


5 

(6%) 


2 

(2%) 


5 

(6%) 


9 

(10%) 


1 


Parents of students with disabilities help make the decision to test 
students out of level In the state tests. 


61 

(69%) 


12 

(13%) 


0 


8 

(9%) 


8 

(9%) 


1 


When selecting students for an out-of-level test, Individualized 
Education Program (lEP) teams consider the future consequences of 
testing students out of level In the state tests. 


30 

(34%) 


16 

(18%) 


6 

(7%) 


29 

(33%) 


8 

(9%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


12 

(13%) 


19 

(21%) 


10 

(11%) 


39 

(44%) 


9 

(10%) 


n. 


Students who need to take the state tests out of level can meet the 
standards of the grade In which they are enrolled in school. 


17 

(19%) 


46 

(52%) 


6 

(7%) 


10 

(11%) 


10 

(11%) 


0. 


Students who need to take the state tests out of level do meet the 
standards of the grade In which they are enrolled in school. 


10 

(11%) 


50 

(56%) 


7 

(8%) 


12 

(13%) 


10 

(11%) 


p- 


Administering the state tests out of level is easy to do. 


50 

(56%) 


14 

(16%) 


7 

(8%) 


9 

(10%) 


9 

(10%) 























































































































Table 19. State 4 -Assessment Coordinator Opinions on Testing Students with Disabilities Out of Level in 
Large-Scale Assessments 



1 


Survey Items 


Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


No 

Response 


a. 


Out-of-level testing is a good way to include students with disabiiities 
in the state tests. 


44 

(81%) 


0 


2 

(4%) 


0 


8 

(15%) 


b. 


Aiiowing students with disabilities to take the state tests out of levei 
has negative consequences for aN students. 


1 

(2%) 


41 

(76%) 


2 

(4%) 


2 

(4%) 


8 

(15%) 


c. 


Aiiowing students with disabilities to take the state tests out of levei 
has negative consequences oniv for students with disabilities. 


0 


44 

(81%) 


1 

(2%) 


1 

(2%) 


8 

(15%) 


d. 


Students who need to take the state tests out of level have more 
accurate test scores than when taking state tests on grade ievei. 


31 

(57%) 


5 

(9%) 


5 

(9%) 


5 

(9%) 


8 

(15%) 


e. 


Students who need to take the state tests out of level are more 
motivated to perform weil than when taking state tests on grade levei. 


30 

(56%) 


5 

(9%) 


4 

(7%) 


6 

(11%) 


9 

(17%) 


f. 


Students who need to take the state tests out of level are less 
frustrated than when taking state tests on grade level. 


38 

(70%) 


3 

(6%) 


2 

(4%) 


3 

(6%) 


8 

(15%) 


g- 


Students who need to take the state tests out of level guess at 
answers less frequently than when taking state tests on grade level. 


33 

(61%) 


3 

(6%) 


1 

(2%) 


9 

(17%) 


8 

(15%) 


h. 


Students who need to take the state tests out of level answer more 
test questions than when taking state tests on grade level. 


34 

(63%) 


4 

(7%) 


0 


8 

(15%) 


8 

(15%) 


1 


Individualized Education Program (lEP) teams can select students 
who need out-of-level tests accurately. 


44 

(81%) 


1 

(2%) 


0 


1 

(2%) 


8 

(15%) 


1 


Individualized Education Prooram flEPf teams actuallv do select 
students who need out-of-level tests accurately. 


40 

(74%) 


3 

(6%) 


0 


3 

(6%) 


8 

(15%) 


1 


Parents of students with disabilities help make the decision to test 
students out of level in the state tests. 


27 

(50%) 


8 

(15%) 


0 


10 

(19%) 


9 

(17%) 


1 


When deciding to test students out of level, Individualized Education 
Program (lEP) teams consider the future consequences of taking the 
state tests out of level. 


16 

(30%) 


9 

(17%) 


4 

(7%) 


17 

(31%) 


8 

(15%) 


m. 


Parents of students with disabilities understand the future 
consequences of taking the state tests out of level. 


7 

(13%) 


11 

(20%) 


2 

(4%) 


26 

(48%) 


8 

(15%) 


n. 


Students who need to take the state tests out of level can meet the 
standards of the grade in which they are enrolled in school. 


7 

(13%) 


30 

(56%) 


1 

(2%) 


8 

(15%) 


8 

(15%) 


0. 


Students who need to take the state tests out of level do meet the 
standards of the grade in which they are enrolled in school. 


3 

(6%) 


32 

(59%) 


1 

(2%) 


9 

(17%) 


9 

(17%) 


p- 


Administering the state tests out of level is easy to do. 


24 

(44%) 


15 

(28%) 


4 

(7%) 


3 

(6%) 


8 

(15%) 
























































































































grade of enrollment standards (item n). And, they did not arrive at a group consensus about lEP 
teams considering the future consequences of testing out of level when making that decision for 
a student (item 1) or parents understanding the consequences of testing their child out of level 
(item m). The assessment coordinators also were unsure whether administering out-of-level 
tests is an easy thing to do (item p). 

We also asked all of the participant subgroups in State 4 what information lEP teams use to 
select the appropriate test grade level at which to administer an out-of-level test to a student 
with a disability. Participants were given a list of options with the directions to select all types 
of information that applied to their school. Responses are displayed in Table 20. Eor all of the 
groups, the student’s instructional level was the most prevalent method for selecting the out-of- 
level test grade level, followed by reviewing the student’s previous test scores and comparing 
the test items to the curriculum the student is being taught in the classroom. Along with a small 
number of participants indicating that their district uses the test company’s locator test scores or 
the student’s standardized achievement test scores, some participants provided another method of 
selection not included in the list. These methods included considering individual student needs, 
the student’s lEP or 504 plan, student performance on other assessments or in the classroom, or 
resulting from a team discussion including special education teachers or case managers. 

Table 20. State 4 - Student Information Used to Choose Out-of-Level Test Grade Levels 





Special Ed 


Regular Ed 


School 


Assessment 


Student Information Used 


Teachers 


Teachers 


Administrators 


Coordinators 


Students’ instructional level 


41 


12 


75 


44 




(85%) 


(63%) 


(84%) 


(81%) 


Students’ previous test scores 


28 


6 


42 


25 




(58%) 


(32%) 


(47%) 


(46%) 


Test company’s locator test 


2 


0 


4 


3 


scores 


(4%) 




(4%) 


(6%) 


Standardized achievement test 


8 


2 


8 


8 


scores 


(17%) 


(11%) 


(9%) 


(15%) 


Test item and curricular 


15 


8 


35 


19 


comparison 


(31%) 


(42%) 


(39%) 


(35%) 


Other 


10 


2 


15 


6 




(21%) 


(11%) 


(17%) 


(11%) 



Narrative Results - State 4 

Three open-ended survey items were completed by the participants in State 4. The subgroups 
were combined across the state for this qualitative analysis with the results presented in Table 
21 . 
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Table 21 . State 4 - Narrative Response to Open-ended Survey Items 



Survey Items 


Most Frequent Result 
Statements 


Supportive Quotes 


(1) How does your district use 
out-of-ievel test scores? 


la. Out-of-level test scores 
were used to monitor and 
evaluate the progress of 
individual students and 
curricular programs. 


“Our school uses all results 
to help Identify the strengths 
and weaknesses of our 
students in order to develop 
meaningful curriculum,” and 
“To demonstrate individual 
growth in student learning.” 




1 b. The use of out-of-level test 
scores was NOT known. 


“Some of us are new to the 
district, so we don’t know.” 




1c. Out-of-level test scores 
were used with students’ 
Individualized Education 
Program. 


“To develop lEP goals and 
objectives” and “To measure 
relevance of a student’s lER” 




1d. Scores were used for 
planning instruction. 




(2) What are the positive 
aspects or impacts of testing 
students out of level? 


2a. Out-of-level testing 
decreased test frustration for 
students with disabilities. 


“Students are less frustrated 
and tend to work harder” and 
“They are more motivated to 
try.” 




2b. Out-of-level testing allowed 
for increased success for 
students. 


“The success they enjoy 
makes them feel a sense of 
achievement.” 




2c. Out-of-testing included 
students with disabilities in 
large-scale assessments. 


“Helps all children to 
participate, meaningfully.” 




2d. Out-of-level testing 
provided a more valid measure 
of student ability. 


“The test scores are more 
accurate.” 




2e. Out-of-level testing 
permitted students to be 
tested at their instructional 
level. 


“All students are tested on 
information that is appropriate 
for them.” 


(3) What are the negative 
aspects or impacts of testing 
students out of level? 


3a. Students with disabilities 
tested out of level feel different 
than other students. 


“Exclusion from standard 
process with social 
consequences,” “Students may 
feel that they are different from 
their peers,” and “Students feel 
singled out as less intelligent.” 




3b. There were problems 
administering out-of-level 
tests. 


“Difficult to administer in large 
schools.” 




3c. Out-of-level test did NOT 
test students with disabilities 
at the grade in which they are 
enrolled. 


“It tests students on how they 
are doing at a grade level in 
which they are not enrolled.” 




3d. There are NO negative 
aspects of out-of-level testing. 
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Discussion 



The purpose of our discussion of these written survey results is two-fold. First, we consider 
the responses across the four states from the open-ended survey items holistically. The purpose 
of this look at our data is to confirm the variety of opinions held by educators in practice that 
have little consistency or uniformity. In other words, the viewpoints do not align easily into 
categories either by educator role or state membership. After we have confirmed these types of 
assumptions about out-of-level testing, we once again take a step back from our data to make 
item by item comparisons to see how the four states that participated in our survey differed 
on these dimensions. Each survey item that garnered numeric data addresses opinions that are 
commonly held in educational practice, but to date have not been grounded in data-based in- 
formation. We follow our discussion of conclusions with a discussion of the explanation of the 
patterns in our data base. 

Narrative Findings - Open-ended Survey Items 

We interpreted our narrative thematic findings across states to draw overarching conclusions 
that deal with all states’ narrative responses. 

There was wide disparity in participants’ opinions about the benefits and disadvantages 
of out-of-level testing. 

In terms of the narrative findings from this survey project, there was no pattern to our data that 
would suggest that some subgroups in our sample held beliefs consistently across the group. 
Rather, there was wide variability in what educators thought were benefits or disadvantages to 
out-of-level testing. For instance, when asked about the positive aspects of testing students with 
disabilities below their grade of enrollment in school, some participants indicated that there 
were no benefits to out-of-level testing while others believed that out-of-level testing is a posi- 
tive means of including more students with disabilities in statewide testing in a less frustrating 
test situation. With decreased stress, some participants thought that students experience more 
success and were better motivated test takers. When tested at the student’s instructional level, 
the resulting test scores were a more valid measure of what students know and can do. 

On the other hand, participants identified negative social experiences for students who are 
embarrassed to be tested below their grade of enrollment when asked about the drawbacks to 
out-of-level testing. Also, learning expectations are lower for students with disabilities who are 
tested out of level that in turn puts them at risk for not receiving a regular high school diploma. 
Others were also concerned that out-of-level testing did not foster achieving grade level content 
standards. Yet, other participants indicated that there are no negative aspects to out-of-level 
testing. 

To look at the patterns of responding by educator role and state membership, there is some 
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overlap. Again, our results do not point to a clear consensus of opinion. In terms of positive 
aspects of out-of-level testing, all four states indicated most frequently that reducing test tak- 
ing frustration was the overall benefit of testing out of level. Beyond that, there was agreement 
about the type of benefit but the frequency at which the benefits were named varied widely. 
Also, State 1 listed a benefit, practice test taking skills that no other state named. State 4 listed 
two benefits, better test performance and increased test score validity, which were not named 
by any other state. 

When comparing states’ responses about the negative aspects of out-of- level testing, the results 
are even more irregular than when naming the benefits of out-of- level testing. States 1 and 4 most 
frequently indicated that negative impacts on students’ self esteem was the biggest drawback 
to out-of- level testing, while States 2 and 3 selected the negative future effects most frequently. 
The second most frequent response for State 1 and 2 was that out-of-level test scores were not 
useful while States 3 and 4 did not mention the lack of usefulness at all. Other factors identified 
as negative aspects were not achieving grade level standards, lowered learning expectations, and 
problems in administering the tests. States 1 and 2 indicated that there were no negative aspects 
to out-of-level testing, but did so the least frequently. 

If out-of-level test scores are useful, they are used for instructional planning. 

Thinking generally about respondents’ ideas about how out-of-level test scores are used, again 
there was wide variability both within and across states. Some participants indicated that out- 
of- level test data were useful for instructional decision making while others thought that out-of- 
level test data were useful developing students’ lEPs. There were also frequent responses that 
out-of-level test scores were neither useful nor used in educational practice. 

The specific responses that respondents generated across states shows some overlap in responses, 
although again a pattern in responding is difficult to discern. States 3 and 4 indicated most 
frequently that out-of-level test scores are used to monitor students’ academic achievement 
while the most frequent response for State 1 was that the scores were not used and for State 
2 to increase participation in statewide testing. Both State 1 and State 2 did mention progress 
monitoring, but not as frequently as the other states. The only response shared by all four states, 
although not at the level of frequency, was that respondents did not know how out-of- level test 
scores were used. 

Narrative Findings - Closed Survey Items 

In addition to the open-ended items that garnered narrative responses, we also presented items 
that required participants to select among a list of narrative responses that were provided for 
them. The item requested participants to select all options that are used in their school district 
to choose the appropriate level for an out-of-level test. Options available were considering a 
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student’s instructional level, considering previous test scores, administering a test company’s 
locator test, administering a standardized achievement test, or comparing test items and a 
student’s curriculum. 

Considering a student’s instructional level was the most frequently selected option used 
to determine a test level at which an out-of-level test would he administered. 

In comparing the four states on this item, there is more agreement across states and subgroups than 
in the responses to other survey items. All four states indicated most frequently that a student’s 
instructional level is the determining factor in selecting a test level at which to administer an 
out-of-level test. As a second most frequent response. States 2 and 3 along with special educa- 
tion teachers in State 4 responded that a student’s previous test scores are used to set a test level. 
Otherwise, all other possible selections for this item did not yield frequent enough responses to 
be able to determine a clear majority. In other words, states and subgroup samples were unclear 
about using the remaining factors for deciding on the test level for an out-of-level test. 

Numeric Findings - Closed Survey Items 

Our second aim in discussing these survey results is to make comparisons on an item by item 
basis to see how states’ results compared and contrasted. We employed our decision rules to 
determine instances where there was a clear majority of opinion and where there was no clear 
majority, from which we then created overarching themes that crossed all states. 

Most participants across states agreed that out-of-level testing was an advantageous way 
to include students with disabilities in statewide testing. 

We first consider item a, whether out-of-level testing is a good way to include students with 
disabilities in statewide testing. With 60% in agreement for State 1 and 83% agreement for State 
4, these two states overwhelmingly affirmed the use of out-of-level testing. While State 3 was 
not in complete agreement with this item, the district test subgroup did agree that out-of-level 
testing was a positive approach to statewide testing. In contrast. State 2 and the special educa- 
tors from State 3 did not agree that out-of-level testing was a good option within a statewide 
testing program. This finding is not surprising given the results of the open-ended survey items 
that pointed to varying opinions as well. 

There were high amounts of disagreement across states as to whether out-of-level testing 
was disadvantageous for students. 

Thinking now about items b and c, whether out-of-level testing has negative consequences for 
all students or just students with disabilities. States 1, 2, 3, and 4 disagreed. Response percent- 
ages of disagreement ranged from 45% to 76% for item b and 50% to 75% for item c indicating 
relatively high amounts of disagreement among participant subgroups across states. Given the 
relatively high level of agreement about item a, the benefits of out-of-level testing as a statewide 
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testing option, this finding too is not particularly surprising. The findings are relatively straight- 
forward in that out-of-level testing is thought to be a “good” testing option that does not have 
negative consequences for either all students or just students with disabilities. 

There was little agreement across states as to whether out-of-level testing is beneficial for 
students. 

Yet, when our survey items began teasing apart the aspects that comprise the first three survey 
items, the consistency in our findings began to breakdown. Survey items d through h deal with 
the various aspects of out-of-level testing thought to be beneficial such as increased test score 
accuracy, better test motivation, reduced frustration, less guessing, and increased test item an- 
swering. However, the responses to these items are not necessarily consistent with the seemingly 
overriding affirmation of out-of-level testing evident in the first three survey items. Except for 
States 1 and 4 where respondents agreed with each benefit most frequently, the other two states 
showed variable responding that contradicted the results of items a, b, and c. State 2 responded 
most frequently that out-of-level test scores were not more accurate than on-grade level test 
scores. There was no clear majority of responses for the remaining items indicating that there 
was no clear agreement or disagreement about the benefits of out-of-level testing. State 3 also did 
not demonstrate clear agreement or disagreement for items d through h, except that respondents 
did reply most frequently that out-of-level testing did reduce student test taking frustration. 

Participants agreed that lEP teams can make accurate decisions about testing students 
out of level, but one state thought that they did so. 

Items i and j requested participants to indicate whether lEP teams can and do select students 
with disabilities appropriately for out-of- level testing. States 1, 2, and 3 overwhelmingly agreed 
that lEP teams can select students appropriately for out-of-level testing, but were uncertain as 
to whether that was actually happening in practice. These results are not as contradictory as 
would first appear when out-of-level testing context is considered. All four states have indicated 
that they provide various training formats on out-of-level testing criteria so that practitioners 
understand how to select students appropriately (Thurlow & Minnema, 2001). Only State 4 has 
specific monitoring practices in place so that all applications for out-of- level testing are reviewed 
by state level personnel. It seems reasonable that only this state would be able to respond con- 
fidently that lEP teams do select students appropriately. 

States believed that parents help make decisions to test out of level in ways that consider 
the future consequences for students, but that parents do not necessarily understand the 
long term ramifications. 

Three items, items k, 1, and m, pertained to decision making about student selection for out- 
of-level testing. Items k and m addressed parent participation in particular, for which we found 
unique patterns in responding. In terms of helping to make the decision to test a student with 
disabilities out of level, all participants in each state indicated that parents helped make the 
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decision. When it came to understanding the future consequences of testing students out of 
level, there was no clear pattern in opinion. In State 1, opinions varied by educator role. School 
administrators agreed that lEP teams considered the future consequences when deciding to 
test out of level, but special education teachers were not sure whether parents understood the 
ramifications. State 2 was not clear about either lEP team considerations or parent understand- 
ings for future consequences. State 3 was similar to State 1 in that responses indicated that lEP 
teams consider future consequences of out-of-level testing when selecting students, but were 
not certain that parents understood those consequences. In State 3 though, responses did not 
vary by educator as they did in State I . State 4 also thought that parents helped make decisions 
about out-of-level testing and the future consequences were considered, but these respondents 
did not know whether parents understood the consequences or not. 

Across all four states, participants believed that students with disabilities who were tested 
out of level could neither be capable of meeting nor do meet grade-level content stan- 
dards. 

Eor items n and o, which address whether students with disabilities can and do meet grade level 
content standards, there was complete consistency within the responses of States 1, 2, and 3. 
None of these states indicated that they thought that students with disabilities who are tested 
out of level are capable of achieving grade-level content standards as indicated by grade level 
proficiency standards. These states also agreed that students with disabilities who were tested 
out of level did not meet grade level standards in their actual classroom learning experience. 

There was no consistent answer in terms of whether administering out-of-level tests is 
easy to do. 

The last closed item in our survey, item p, asked participants to indicate whether out-of-level 
tests were easy to administer. Opinions were not necessarily clearly delineated. State 1 and 
one subgroup of participants in State 3 (district test coordinators) were not sure whether the 
process of testing students with disabilities out of level was easy to do. On the other hand. State 
2 indicated that administering out-of-level tests was difficult to do while State 4 indicated that 
the administration was easy to do. 



Conclusions 

To conclude the reporting of this survey research, we discuss three broad themes that emerge 
from our consideration of the narrative and numeric data as a composite whole. These themes 
are written generally enough to provide thoughtful considerations for all stakeholders, but cannot 
be generalized specifically to other educator subgroups or states that test out of level because 
we used a purposive sample from which to collect our data. 
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Educators’ positive perceptions of out-of-level testing change when the testing option is 
thought about with greater detail. 

Taken all together, our findings suggested that when educators are thinking generally about 
out-of-level testing, the option appears to be a reasonable solution for including students with 
disabilities in statewide testing. When asked if out-of-level testing was an advantageous way to 
include students with disabilities in large-scale assessments, a majority of participants indicated 
this to be true. Also, when asked if out-of-level testing had negative consequences for students 
with and without disabilities, participants basically disagreed. 

However, when prompted to think about out-of-level testing on a deeper level by considering 
specific aspects of this approach to testing, inconsistencies began to emerge. For instance, while 
all states agreed overall that out-of- level testing did reduce student frustration during test taking, 
other benefits of out-of-level testing were not as readily agreed on. Within the results for the 
remaining benefits, there was wide variability in terms of both agreement and disagreement. 

Even more confusing is the conflicting responses on different survey items. For example, the 
response pattern clearly indicated that participants’ agreed that students with disabilities who 
were tested out of level could not meet grade level content standards. In looking at the thematic 
results from the narrative data, participants in two states indicated that students who are tested 
out of level are unable to receive a high school diploma. This negative aspect of testing students 
below the grade in which they were enrolled in school seems to be in direct contrast to the fact 
that these students cannot achieve on-grade level. As another example, all participant subgroups 
disagreed that out-of- level testing has negative consequences for students with disabilities (item 
c) with responses ranging from 50% to 81% disagreement. When asked in an open-ended item 
to generate negative aspects or impacts of out-of-level testing on students with disabilities, all 
subgroups across all four states produced multiple negative aspects or impacts. This narrative 
finding that prompted respondents to think more deeply about out-of-level testing is in direct 
contrast to the results from the closed item, which was written more generally. 

Educators believe that students with disabilities who are tested out of level cannot meet 
content standards set for their grade of enrollment. 

Possibly the most discouraging aspect to our survey findings was the uniformity in responses 
regarding students with disabilities who have been tested out of level and their learning expecta- 
tions. All sample subgroups, which included special and general educators, administrators, and 
district test coordinators, indicated that those who participate in out-of-level testing could not 
and do not meet grade level standards. This finding is particularly alarming given the federal 
mandate in NCLB that all students meet grade-level proficiency on states’ content standards. 

In today’s accountability minded environment, it is essential that schools figure out ways that all 
students can achieve at optimum levels. Looking at instructional issues, test instrument access, or 



NCEO 



33 





accommodations use is a place at which planning can begin. The consequences are much larger 
than learning and test taking decisions that occur on a daily basis. Large-scale assessment data 
that are used for accountability purposes have far reaching implications for students, teachers, 
administrators, and schools. A subgroup of a school’s population loses valuable opportunities to 
learn while teachers and administrators may be blamed for lack of demonstrating proficiency on 
grade level content standards. School improvement planners are not given a true picture of how 
all students are performing so that their decisions do not improve learning for every student. 

Educators’ opinions about out-of-level testing vary widely so that consensus is rare. 

The analysis of both numeric and narrative data from our survey yielded results that were highly 
variable across states and subgroups of educators. Agreement occurred about using students’ 
instructional levels to determine the test level of an out-of-level test, that lEP teams can select 
students accurately for out-of-level testing, that parents help make the decision to test below 
grade level, and that students with disabilities who are tested out of level cannot and do not 
meet the content standards for their grade of enrollment. The often disparate opinions held by 
practitioners suggest that they have not coalesced on a common set of viewpoints about testing 
students with disabilities out of level in large-scale assessment programs that are used for ac- 
countability purposes. Nevertheless, their general impressions are clear - out-of-level testing 
is more frequently seen in a positive light than a negative one. 

The debate will undoubtedly continue concerning the value of testing students with disabilities 
below their grade of enrollment in school. Given our recent findings (Minnema et ah, 2004a; 
Minnema et ah, 23004b; Minnema & Thurlow, 2003; Thurlow et ah, 2003), it behooves all 
stakeholders to understand that the passion with which opinions are sometimes expressed may 
not represent all educators’ opinions. For each discussion of out-of-level testing, we recom- 
mend thinking through the issues from multiple perspectives. It is through critical thought that 
we understand the deeper issues that surround contentious educational conundrums such as 
out-of-level testing. 
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Appendix A 

Survey of Administrators, Teachers, and District Test Coordinators 



SURVEY OF [educator role] 

Out-of-Level Testing in [state name] 

Definition; “Out-of-level testing” means giving a student the [state test name] at a level lower 
than the student’s assigned grade level. 

Ql. Are you familiar with out-of-level testing? 

1 . Yes (Please continue with the survey.) 

2. No (Your survey is complete. Please return it to the registration table.) 

Q2. To what extent do you agree or disagree with the following statements? {Circle 
one answer for each item.) 



1 




Agree 


Disagree 


No 

Opinion 


Don’t 

Know 


a. 


Out-of-level testing is a good way to include students with disabilities in the 
[state test name]. 


1 


2 


3 


4 


1 


Allowing students with disabilities to take the [state test name] out of level has 
negative consequences for aU students. 


1 


2 


3 


4 


c. 


Allowing students with disabilities to take the [state test name] out of level has 
negative consequences only for students with disabilities. 


1 


2 


3 


4 


d. 


Students who need to take the [state test name] out of level have more accurate test 
scores than when taking the [state test name] on grade level. 


1 


2 


3 


4 


e. 


Students who need to take the [state test name] out of level are more motivated to 
perform well than when taking the [state test name] on grade level. 


1 


2 


3 


4 


f. 


Students who need to take the [state test name] out of level are less frustrated than 
when taking the [state test name] on grade level. 


1 


2 


3 


4 


g- 


Students who need to take the [state test name] out of level guess at answers less 
frequently than when taking the [state test name] on grade level. 


1 


2 


3 


4 


■ 


Students who need to take the [state test name] out of level answer more test 
questions than when taking the [state test name] on grade level. 


1 


2 


3 


4 


i. 


Individualized Education Program (lEP) teams can select students who need out-of- 
level tests accurately. 


1 


2 


3 


4 
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j- 


Individualized Education Proeram QEPI teams actuallv do select students who need 
out-of-level tests accurately. 


1 


2 


3 


4 


1 


Parents of students with disabilities help make the decision to test students out of 
level in the [state test name]. 


1 


2 


3 


4 


1. 


When deciding to test students out of level, Individualized Education Program (lEP) 
teams consider the future consequences of taking the [state test name] out of level. 


1 


2 


3 


4 


m. 


Parents of students with disabilities understand the future consequences of taking the 
[state test name] out of level. 


1 


2 


3 


4 


n. 


Students who need to take the [state test name] out of level can meet the standards of 
the grade in which they are enrolled in school. 


1 


2 


3 


4 


o. 


. Students who need to take the [state test name] out of level do meet the standards of 
the grade in which they are enrolled in school. 


1 


2 


3 


4 


p- 


Administering the [state test name] out of level is easy to do. 


1 


2 


3 


4 



Q3. Which of the following do teachers in your district use to choose the appropriate 
level for an out-of-level test? ( Circle all that apply.) 

a. Consider a student’s instructional level 

b. Consider a student’s previous test scores 

c. Administer a test company’s locator test 

d. Administer a standardized achievement test 

e. Compare test items and a student’s curriculum 

f. Another way (Please specify) 



Q4. How does your district use out-of-level test scores? 



Q5. What are the positive aspects or impacts of testing students out of level in the [state 
test name]? 
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Q6. What are the negative aspects or impacts of testing students out of level in the [state 

test name]? 



Q7. Do teachers in your district test students with disabilities out of level in the [state 
test name]? 

1. Yes 

2. No 



Q8. How long have you worked in the field of education? years 



Q9. What is the highest degree you have obtained? {Circle one.) 

1. Bachelor’s degree 

2. Master’s degree 

3. Specialist’s degree 

4. Ph.D. 



QIO. What is your position in your school system? 



Please return your survey to the registration table to receive a report from the National 
Center on Educational Outcomes! 



NCEO 



39 





